DOCUMENT RESUME 



ED 423 244 



TM 028 973 



TITLE 



INSTITUTION 

REPORT NO 
PUB DATE 
NOTE 

PUB TYPE 
EDRS PRICE 
DESCRIPTORS 



IDENTIFIERS 



Student Testing: Issues Related to Voluntary National 
Mathematics and Reading Tests. Report to the Honorable 
William F. Goodling, Chairman, Committee on Education and 
the Workforce, House of Representatives, and the Honorable 
John Ashcroft, U.S. Senate. 

General Accounting Office, Washington, DC. Health, 

Education, and Human Services Div. 

GAO/HEHS- 98 - 163 
1998-06-00 
4 5p . 

Reports - Evaluative (142) 

MF01/PC02 Plus Postage. 

* Agency Cooperation; Costs; ♦Educational Testing; Elementary 
Education; Elementary School Students; Grade 4; Grade 8; 
Mathematics Achievement; ^National Competency Tests; Reading 
Achievement; *State Programs; Student Characteristics; 
Student Evaluation; *Test Construction; Test Results; 
♦Testing Programs 

Department of Education; National Assessment Governing 
Board; National Assessment of Educational Progress; 
♦Voluntary Participation 



ABSTRACT 



Information was sought about three main issues related to 
voluntary national tests for fourth-grade reading and eighth-grade 
mathematics: (1) the relationship between the National Assessment Governing 

Board and the U.S. Department of Education; (2) costs of developing the 
national voluntary tests and procedures for hiring contractors; and (3) 
possible explanations for the differences in scores on state achievement 
tests and National Assessment of Educational Progress (NAEP) tests. When 
Congress gave the NAEP Governing Board authority for voluntary test 
development, it altered the relationship between the Governing Board and the 
Department of Education. Department of Education assistance was not used in 
developing the test contract, as had been typical of NAEP tests in the past. 

A review of the test development contract suggests that the cost to the 
federal government for developing one complete set of tests would be $15 
million, and could go as high as $96 million for a complete set of tests for 
grades 4 and 8 if all students participate. A number of explanations have 
been proposed for the differences between state and NAEP results, but most 
center on differences in definitions of proficiency, differences in the kinds 
of tests administered, and differences in the kinds of students taking the 
tests. Appendixes present the study questions, a memorandum of understanding 
between the Department of Education and the Governing Board, the NAEP 
response to outside recommendations, and comments from both agencies. 
(Contains three tables and one figure.) (SLD) 
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June 18, 1998 

The Honorable William F. Goodling 

Chairman, Committee on Education and the Workforce 

House of Representatives 

The Honorable John Ashcroft 
United States Senate 

In February 1997, the Clinton administration announced a plan to develop 
voluntary national tests for fourth grade reading and eighth grade 
mathematics. The tests were proposed as a way to provide parents and 
teachers with information about their students’ performance relative to 
widely agreed-upon standards of what students should know and be able 
to do. Concerns have been raised, however, about the need for such tests 
and the potential for inappropriate federal influence on school curriculum, 
which is established by the states and localities. 

Originally, the Department of Education had complete responsibility for 
the testing initiative. In November 1997, however, the Department’s 
Appropriations Act transferred responsibility for developing the tests from 
Education to the National Assessment Governing Board (referred to as the 
“Governing Board”), 1 a board originally created as part of the Education 
Department in 1988 to set policy for a program called the National 
Assessment of Educational Progress (naep). In addition, the Congress 
directed the Department, including the Governing Board, to use no fiscal 
year 1998 funds to, among other things, pilot test, field test, or administer 
these tests. 

The proposed voluntary national tests are designed to be based on naep. 

As part of your ongoing review of the proposed testing program, you asked 
us to provide you with information about three main issues: 

• the relationship between the Governing Board and the Department of 
Education, 

• costs of developing the national voluntary tests and procedures for hiring 
contractors, and 

• possible explanations for the differences in scores on state achievement 
tests and naep tests. 



'Departments of Labor, Health, Human Services, and Education and Related Agencies Appropriations 
Act, 1998. P.L. 105-78, sec. 307 (a), 111 Stat 1467, 1606 (1997). 
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To address the first two issues and other specific questions (see app. I), 
we interviewed officials at both the Governing Board and the Department 
of Education, reviewed contract files, and analyzed contracts and other 
relevant documents. To determine reasons for differences between state 
and naep test results, we interviewed testing experts with the Governing 
Board, the Department, and the National Center for Research on 
Evaluation Standards on Student Testing and reviewed selected literature 
on testing standards. We also observed a Governing Board meeting and 
explored whether outside advisers to the voluntary national test 
specification process had written textbooks or educational materials for 
elementary and secondary education classrooms. We conducted our work 
between January and May 1998 in accordance with generally accepted 
government auditing standards. 



Results in Brief 



Since 1992, the Governing Board and the Department of Education have 
operated under a Memorandum of Understanding based on naep’s 
authorizing legislation and designed to ensure the Governing Board’s 
independence from its parent agency in fulfilling its responsibilities for 
naep, a national assessment program. In accordance with this 
memorandum, the Governing Board establishes program policies and 
standards, oversees contracts to develop test specifications, makes its 
own personnel decisions, obligates funds, and awards contracts. Because 
they share responsibilities for the program, however, the Department and 
the Governing Board collaborate extensively. Although the Governing 
Board helps in developing the Department’s budget request for naep- 
related work, the Department has final authority over the request 
forwarded to the Office of Management and Budget (omb). 

When the Congress gave the Governing Board exclusive authority for the 
voluntary national test development contract in November 1997, it also 
altered the relationship between the Governing Board and the Department 
regarding this contract. Because it now had exclusive authority for this 
development contract, the Governing Board used no Department 
assistance in reviewing the contract as it had typically done for naep- 
related contracts. As a result of its review, the Governing Board 
substantially changed the test development contract. For example, under 
the Department’s contract, the planned administration date of the first test 
was March 1999; under the Governing Board’s contract, the planned 
administration date is March 2001. Another modification specified that the 
Governing Board, not the Department, would make all policy decisions as 
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specified in the original contract. Nor did the Governing Board help in 
developing the Department’s budget request for voluntary national test 
development funds. 

On reviewing the Governing Board’s test development contract, we 
estimated that the cost to the federal government for developing one 
complete set of tests would be $15 million. The cost of implementing the 
fourth grade reading and eighth grade mathematics testing program has 
been estimated at up to $96 million if all fourth and eighth grade students 
in public and private schools participated. Who would pay for 
implementing the test has not been determined. For fiscal years 1997 and 
1998, contracts awarded for voluntary national tests totaled about 
$17 million. Most of this amount — about $13 million — was for the 1997 
contract for developing these tests; however, only about $1.7 million of 
this contract award had been paid to the contractor as of April 1998. In 
keeping with the congressional mandate to restrict the use of fiscal year 
1998 funds, the only contract awarded with these funds was for a new 
congressionally mandated study by the National Academy of Sciences 
(nas), which included examining the feasibility of comparing commercial 
and state tests with naep tests. In addition to those expenditures for which 
cost fig ures were available, the Department also spent an undetermined 
amount on distributing information about the national tests in fiscal years 
1997 and 1998 by using web sites, press releases, articles in trade 
magazines, Department newsletters, and public presentations. Department 
officials said these dissemination activities are not directly related to 
developing the test and were part of the Department’s ongoing function to 
educate the public about its program initiatives. 

We heard various explanations for the differences in scores on the naep 
and state achievement tests; far fewer students achieve proficiency on 
recent naep tests than on state tests. Most explanations of the differences 
fell into three main categories: (1) differences in how the tests define 
proficiency and determine the threshold test score for proficiency (for 
example, one test may define proficiency as getting 66 percent of all items 
correct; another might define proficiency as scoring a minimum of 
66 percent on each test section); (2) differences in the kinds of tests 
administered (for example, multiple choice versus problem solving); and 
(3) differences in students taking the test and when they take it. naep, for 
example, tests students with disabilities or limited-English proficiency; 
some state tests may not. 
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Background 



The Governing Board and 
the Department Share 
Responsibility for NAEP 



Created by the Congress in 1988, the Governing Board provides 
independent, bipartisan policy guidance for naep, a congressionally 
authorized program in place since 1969. The Secretary of Education 
appoints the 25 voting members of the Governing Board to represent a 
wide range of legislatively prescribed categories, including two state 
governors from different political parties, a school board member, a 
school district superintendent, classroom teachers, a business or industry 
representative, and curriculum specialists. The Department’s Assistant 
Secretary for Educational Research and Improvement serves as a 
nonvoting, ex officio member. 

The Governing Board has joint responsibilities with the Department of 
Education for naep. The Governing Board sets policy for naep, and the 
Department’s National Center for Education Statistics (nces) oversees the 
development and implementation of the assessment according to the 
Governing Board’s guidance. The nces commissioner is responsible for 
naep operations and technical quality control. In November 1997, the 
Congress gave the Governing Board exclusive authority— separate from 
the Department — for the voluntary national test development contract. 



naep primarily tracks the educational achievements of students in grades 
4, 8, and 12 in selected subject areas such as reading, writing, 
mathematics, and U.S. history. 2 The Governing Board and the Department 
share responsibility for developing and implementing each test. 

For example, the Governing Board selects the subjects for each 
assessment cycle 3 and oversees the creation of the conceptual frameworks 
for the tests and the specifications for developing the tests. The 
frameworks identify subject-specific content areas and specify such test 
attributes as the cognitive skills to be tested and the type and number of 
test items in a particular content area. The frameworks are developed 
through a national consensus process that involves educators, 
academicians, policymakers, the general public, and others, nces in turn 
hires contractors who use these frameworks to develop the actual 
assessments (which includes writing and pilot testing the test items and 
field testing the test booklet) and administer them. The Governing Board 



2 Thus far, the Governing Board has overseen the creation of assessment frameworks for reading, 
writing, mathematics, civics, arts, science, geography, and U.S. history. More frameworks for 
additional subject areas are planned. 

3 Since 1990, various state and national NAEP assessments have been administered every 2 years, 
according to a schedule determined by the Governing Board. 
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